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ABSTRACT 



The U.S. Office of Education is undertaking a program of research and 
development designed to redch the following objective: 100% of all persons not 

in permanent care institutions must pass, by age 10, a criterion-referenced 
test which is predictive of competent performance on a set of adult reading tasks 
selected to have favorable returns to the individual and to society in general. 

The purpose of this Request for Proposal is to implement the recommendat ions of 
expert advisors that the objective statement be further specified and 
operationalized through the following activities: (l) definition and description 
of the recognized subgroups of subjects who comprise the target population of the 
program by providing explicit inclusion-exclusion criteria and identifying 
parameters and parameter values which differentiate the subgroups, (2) identi- 
fication of a set of adult criterion reading tasks which adequately sample the 
tasks for which highly favorable returns to the individual and to society can be 
demonstrated and the construction of an assessment procedure to validate the 
choice of those adult reading tasks as the performance criterion dimension of 
the program objective, and (3) determination of resource ceilings within which 
"“instructional systems generated through the targeted R&D program on reading must ~ 
operate. (CK) 
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Foreword 



The National Center for Educational Research and Development views the 
Targeted R&D Program on Reading primarily as an attempt to assemble the 
scientific and engineering expertise necessary for reaching the Right 
To Read Goal, but we also view the Program as a study of research manage- 
ment. For both these reasons, we feel that it is appropriate to document 
every major event in the general literature so that scientists, developers, 
educators, and managers may understand the Program and advise us on how 
to improve it. 

The following scope of work statement was used in awarding the Phase I 
contracts; in its present form, it must not be regarded as a new solici- 
tation. 

Other documents which relate to the Targeted R&D Program on Reading are: 

1. Gephart, William J., Application of the Convergence 
Technique to Basic Studies of the Reading Process. 

(Bloomington, Ind: Phi Delta Kappa, Inc., 1970) 

Final Report of NCERD Project 8-0737. 

ERIC Document 037587; $1.00 in Microfiche; $12.45 
in hard copy. 

2. Carrese, Louis M. and Carl G. Baker, "The Convergence 

Technique: A Method for the Planning and Programming 

of Research Efforts," Management Science, Vol. 13 
No. 8, April 1967. 

3. Gephart, William J. and Monte Penney, "The Convergence 

Technique for Research in the Social Sciences: An 

Application to the Study of Reading," in Reading: 

Process and Pedagogy: Nineteenth Yearbook of the 

National Reading Conference, (Milwaukee: Marquette 

University Press, in press). 

4. Penney, Monte and Richard B. Adams, "Forecasts of 
Future Reading Research," in Bateman, Barbara (ed.) 

Learning Disorders. Vol. 4: The Reading Problem. 

(Seattle: Special Child Publications, scheduled 

for publication in spring, 1971.) 

5. Penney, Monte, Howard F. Hjelm, and William J. Gephart, 

"The Targeted R&D Program on Reading," American Educa- 
tional Research Journal , May 1970. 

6. Penney, Monte, "An Introduction to the Convergence Technique," 
Educational Researcher (Newsletter of the American Educational 
Research Association), February 1970. 

7. Penney, Monte, "Looking Deep into the Reading Process," 

American Education. (Washington: U.S. Office of Education, 

March 1970) 
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8. Gephcrt, William J., "The Targeted Research and 

Development Program on Reading: a report on the 

application of the Convergence Technique," Reading 
Research Quarterly , Summer, 1970. 

9. Ellson, Douglas G. , "A Critique of the Targeted Research 
and Development Program on Reading," Reading Research 
Quarterly , Summer, 1970. 

The final pages of this document are a USOE press release which announced 
the Phase I contract awards. 



Monte Penney 
Research Associate 
National Center: for Educational 
Research and Development 
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(Attachment B - continued) 

IB - Possible Division o€ Activities Among Contractors 

This Request for Proposal solicits work which will implement 
Phase I of the Targeted Research and Development Program on Reading. 

Three "Projects" are described in the Request for Proposal, and each 
"project's" scope of work has three sub-parts. This arrangement has 
been selected for clarity of presentation and for management of the 
review process . 

No single proposal may address more than one of the three major 
Projects. For example, an offeror wishing to compete for sub-part 3 
of Project I and for all of Project II must submit two separate 
proposals. 

Within each major Project, offerors may wish to bid on sub-parts within 
the following guidelines : 

Project I: Refinement of the Program Objective 

Sub-parts 1 and 2 of this Project must be conducted by one 
contractor; sub-part 3 may be conducted by a separate con- 
tractor. An offeror may submit one proposal to bid on all 
sub -parts. 

Project II: Literature Search 

Sub-parts 1, 2, and 3 of this Project must be conducted by one 
contractor; that is, any proposal for Project II must include 
all three sub -parts. 

Project III: Status Survey 

The sub-parts of this Project may be conducted by separate 
contractors. An offeror may submit one proposal to bid on 
one, any two, or all three sub-parts. 



Proposals must be labeled to indicate precisely which Project 
they address . 
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(Attachment B - continued) 

IC 

Targeted R&D Program on Reading 
Project No. I: Refinement of the Program Objective 

I. Statement of Purpose: 

The U.S. Office of Education Is undertaking a program of research 
and development designed to reach the following objective: 1007. of all 

persons not In permanent care Institutions must pass, by age 10, a cri- 
terion - referenced test which Is predictive of competent performance on 
a set of adult reading tasks selected to have favorable returns to the 
Individual and to society In general. 

The purpose of this Request for Proposal Is to Implement the 
recommendations of expert advisors that the objective statement be further 
specified and operationalized through the following activities: 

1. Define and describe the recognized subgroups of subjects who 
comprise the target population of the program by providing 
explicit Inclusion-exclusion criteria and Identifying parameters 
and parameter values which differentiate the subgroups. 

2. Identify a set of adult criterion reading tasks which adequately 
sample the tasks for which highly favorable returns to the 
Individual and to society can be demonstrated and construct an 
assessment procedure to validate the choice of those adult 
reading tasks as the performance criterion dimension of the 
Program objective. 

3. Determine resource ceilings within which instructional systems 
generated through the Targeted R&D Program on Reading must 
operate. 
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II. Definitions : 

1. Assessment procedure: A test instrument which is not subject 

to the cost-efficiency or time-efficiency contraints normally 
required of a fully developed instrument. Specifically, a pro- 
cedure which distinguishes between adults who can perform the set 
of high-benefit reading tasks and those who cannot. 

2. Reading task: A real-life incident which creates an internally 

or externally imposed requirement for an individual to perform 
a discrete, observable operation which is highly dependent upon 
his having satisfactorily read a specific passage of written 
material . 

Examples of reading tasks are: (a) looking up a telephone number; 

(b) following written directions which tell how to assemble a toy 
or an appliance; (c) responding to ~ written social invitiation; 
and (d) completing a written job application. 

3. Class of reading tasks: An aggregation of discrete reading tasks 

according to a logical or empirical classification scheme. 

4. Criterion task: A reading task which represents a class of reading 

tasks. Presumeably , the assessment procedure will be composed of 
criterion tasks. 

5. Adult: A person who (a) is over 16 years of age and (b) makes his 

own major day-to-day decisions on problems such as choo? 'ng a place 
of residence, choosing a job, managing his money, etc. 




(Attachment B - continued) 



3 



III. Scope of Work ? 

1. Define and describe the recognized subgroups of subjects who comprise 
the target population of the program by providing explicit Inclusion- 
exclusion criteria and Identifying parameters and parameter values 
which differentiate the subgroups. 

This activity answers the question, "Who will be able to read when 
the program has satisfied its objective?" 

As originally delineated by the planning project, the target popula- 
tion Includes all persons over 10 years of age who do not have physical 
or emotional handicaps which require permanent institutional care. 

On a logical basis, there may well be people in permanent care insti- 
tutions who can learn co read (e.g., the bright but emotionally 
disturbed child); similarly, persons who are not permanently institu- 
tionalized may include some who cannot learn to read in the usual 
sense of the word (e.g., the well-adapted, self-sufficient blind 
person) . The Contractor shall provide one hundred copies of a report 
which provides (a) inclusion and exclusion criteria for the target 
pcpulation; (b) definition and description of the major subpopulations 
within the target population; and (c) a rationale for the exclusion of any 
category of persons from the target population. 

2. Identify a set of adult criterion reading tasks which adequately 
sample the tasks for which highly favorable returns to the individual 
and to society can be demonstrated and construct an assessment pro- 
cedure to validate the choice of those adult reading tasks as the 
performance criterion dimension of the Program objective. 




This activity requires pursuit of the following general strategy: 
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(a) Design an adequate plan for representative sampling 
of the adult population of the United States. (See 
definition of "adult," above.) 

(b) Identify the universe of reading tasks actually performed 
by the resulting sample. 

(c) Develop one or more schemes for classifying the universe 
of reading tasks. * 

(d) Perform studies to determine the benefits Which accrue to 
the individual and to society when adequate performance 
of a class of tasks can be demonstrated. 

(e) Select criterion tasks which represent high-benefit 
classes of reading tasks. 

(f) Construct an assessment procedure based on the work per- 
formed in (a) through (e) . 

(g) Validate the assessment procedure by testing hypotheses 
approximating the following form: 

"The benefits identified in (d) for a class of reading 
tasks accrue to individuals who can perform the criterion 
tasks which represent that class." 

The Contractor shall provide 100 copies of a report which 
(a) details the results and methodology of his pursuit of 
the general strategy outlined above and (b) provides data 
to permit evaluation of the adequacy of his performance of 
each step in the overall activity. 

*Such schemes would classify reading tasks, on such bases as readability, 
similarity, or frequency of occurence. 
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3. Determine resource ceilings within which reading instruction 
systems generated through the Targeted R&D Program on Reading 
must operate; 

This activity attempts to answer the question, "What is society 
willing and able to invest, in terms of per pupil resource 
expenditure over the 10-year educational period implied by the 
Program objective, to obtain the performance specified in (2)?" 

The results of this study will specify the cost-efficiency 
dimension of the overall objective of the Targeted R&D Program 
on Reading. The Contractor shall supply 100 copies of a report 
detailing his methodology and findings in establishing the cost 
ceiling to be applied. 

IV. Criteria for the Evaluation of Proposals: 

1. Evidence that the Offeror can provide the overall managment capability 
necessary to insure quality, completeness, and efficient utilization 
of time and funds. 

2. Appropriateness of the training and experience of the staff and 
consultants. 

3. Appropriateness and adequacy of the technical methodology proposed 
for each of the three activities in the section titled "Scope of 
Work:" 

4. Evidence that the Offeror has access to, and will seek, advice from 
professional persons and institutions interested in adult reading 
competence. The Offeror's success in this regard provides public 
and professional credibility for his project and for the entire 



program, 
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Targeted R&D Program on Reading 
Project No. 2: Literature Search 

I. Statement of Purpose : 

Preliminary work by the planning group for the Targeted R&D Program 
on Reading has identified three concurrent activities to serve as a basis 
for implementing a large scale R and D effort. One of these is a search 
or searches of the published literature on the phenomena listed below: 

1. the reading process, 

2. the learning- to-read process, and 

3. language development related to reading. 

For each of the phenomena listed above: 

a. Identify existing models or partial models.* 

b. Describe and analyze the models identified. 

c. Combine partial models where logically feasible. 

d. Specify research efforts needed to test and develop 
the models further. 

II. Definitions : 

The phenomena defined below are all under active investigation. 
Accordingly, the research literature contains many alternative definitions 
of them, derived from different disciplinary points of view. The following 
definitions, prepared by the planning team, do not attempt to select the 
"best" alternative. Instead, they are intended to describe their respective 




* Hereafter the term models will also Include partial models 
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phenomena in a global manner appropriate to a multidisciplinary R&D 
program. 

1. Model : A model is a representation of a phenomenon which 

displays the identifiable elements of the phenomenon and the 
relationships among those elements, and the processes through 
which those elements interact. Models serve three general 
purposes: (a) to explain what a complex phenomenon is; (b) 

to describe how the phenomenon works; and (c) to provide the 
basis for predicting the changes which will occur in one element 
of the phenomenon when changes are made in one or more other 
elements. 

2. Partial Model : Models may be partial in either or both of two 

senses: (a) they may attempt to represent only a part of their 

target phenomenon; or, (b) they may be only partially developed, 
given successful prediction as described in (lc) above as the 
criterion for complete development. 

3. Reading Process : The collection of real events which occur 

in a reader while he is engaged in reading behaviors. These 
events probably include physiological ones (neural, bio-chemical, 
etc.) and psychological ones (cognitive, perceptual, affective, 
etc . ) . Reading behaviors are covert responses to written verbal 
language; they are indicated by overt performances which could 
not have occurred without the covert responses. 
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4. Learning- to-read process : The collection of real events which 

occur in an individual in response to formal reading instruction, 
informal reading instruction (e.g|J, random exposure to written 
verbal language in the environment) , and other classes of 
stimuli necessary (but not now specifiable) to the development 

of reading abilities. 

5. Other Language development related to reading ; The acquisition 
of speaking, listening, and writing abilities during the 
acquisition of reading abilities. In general, children are 
able to speak and to listen before they become able to read 
and write. As the latter two abilities develop, they begin 

to affect the former two and vice versa . 

III. Scope of Work ; 

The end product of the work will be a written report or reports which 
include but are not necessarily limited to the items below. The full 
insight of the contractor (s) is sought. 

a. Models of the reading process. 

b. Models of the learning-to-read process. 

c. Models of language development related to reading. 

For each of the above, the contractor will identify all existing models 
and related research reports in the literature and evaluate them on the 
basis of criteria developed by the contractor and accepted by a review 
panel selected jointly by the contractor and USOE, reporting all models 
that satisfy the criteria. For each model, the contractor will describe 
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the key features, identify and analyze all supporting assumptions and 
hypotheses. Further, the report will synthesize all models having 
common structural and/or process elements, making explicit the logical 
bases for each synthesis. The contractor will maintain complete records 
of sources and systematic files of raw data for later input into the 
Targeted R&D Program on Reading information system. 

IV. Plan of Operation ; 

The attached flow chart and explanatory key suggest the level of effort 
that will be required in carrying out the project (s). The attached 
Occasional Paper Number 3, "Profiling Educational Research" by William J. 
Gephart, shows a preferred procedure for activity 23 on the flow chart. 
Bidders are expected to present a documented logical case to justify the 
procedures that the propose to follow, whether those procedures .adopt, 
modify, or replace those shown in the flow chart and occasional paper. 
Regardless of the procedures proposed, bidders will be expected to provide 
a comprehensive listing of information sources that will be used. 

It is expected that the Contractor (s) will find it desirable to meet 
with appropriate OE staff and the review panel at appropriate times. 

On the expiration date of the contract, the contractor will submit to 
the Office of Education 100 copies of the final report in compliance with 
procedures outlined in the NCERD publication, Preparing Research Reports 
for the U.S. Office of Education . The contractor should also be prepared 




to give an oral presentation of the results of the study to an audience 
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selected by NCERD. It is expected that all work will be completed on 
or before June 30, 1971. 

V. Criteria for Evaluation of Proposals : 

Proposals will be evaluated on the basis of the following criteria: 

a. Comprehensiveness of the proposed plan to identify and 
obtain research literature which discusses the reading 
process, the process of learning to read, and language 
developmet ' related to reading. 

b. Quality of the strategy for analysis and evaluation outlined 
and comprehensiveness of the proposed evaluative criteria and 
tools to be used in the assessment of the research literature. 

c. Evidence of ability to manage the overall project. 

d. Adequacy of time and appropriateness of experience and quali- 
fications of institutional and consultative personnel related 
to the project: 

1. Project director 

2. Other key personnel 

3. Supporting staff 

4. Consultants 

e. Availability of resources for collecting and processing 
information. 

f. Economic efficiency 

g. Feasibility of proposed time schedule 
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PROFILING COMPLETED RESEARCH 

The evaluation of the quality of completed research in education 
has two distinct components. The first of these components is the 
problem attacked in the study. The second is the methodological ade- 
quacy of the study conducted. Profiling* the procedure described in 
this paper* deals with the latter* the evaluation of the methodological 
adequacy. It avoids evaluation of the problem itself on the belief 
that the importance of a given problem can only be established through 
an historical perspective. To assert otherwise would imply the exist- 
ence of a preferred value system. 

Evaluation of methodological adequacy of a given piece of research 
is a prerequiste for the acceptance or rejection of the conclusions 
of that investigation. Such conclusions can be no stronger than the 
methods utilized in generating and analyzing the data on which the con- 
clusion is reached. In the past we have operated on the assunption: 
if the methodology is sound* the conclusion can be accepted and vice 
versa. The faultiness of this assumption is one of the problems 
that have long plagued both the improvement of and use of educational 
research. 

Research methodology is multifaceted. It involves an inherent 
logical argument* the selection of subjects to be studied* structuring 
of experiences for those subjects* measurement* and the analysis of the 
generated data. It is possible to have sound procedures in some of 
these facets and weak procedures in others; a possibility that pre- 
cludes a statement that a conclusion is based either on sound or 




unsound methods 
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The problem is further oomplioated. Needs for surety in varying 
times and professional oircumstanoes set the quality standard for re- 
search methods* If the need for knowledge in an area is great, the 

k 

'methodological development orude, and the amount of risk to personal 
safety low, oonolusions can be aooepted and operated on despite weak- 
nesses in their methodological base* In another set of oircumstances 
this would be wholly unacceptable* Since the use to which a conclu- 
sion might be put cannot be controlled, an absolute level of quality 
cannot be established for each research effort* 

Regardless of the knowledge needs or professional circumstanoes, 
a given conclusion ought not to be accepted, held tentatively or re- 
jected without evaluation of the research methods underlying it* It 
is asserted that the profiling procedure described in this paper will 
facilitate the labeling of the methodology of completed research reports. 
When this labeling has been completed, the user of that study can make 
sounder decisions regarding the acceptance or rejection of its conclu- 
sions. 



ELEMENTS IN PROFILING 

In conducting an empirical study an investigator does numerous 
things. Those things are the elements on which the profiling activity 
focuses. They include: (l) the structuring of a logical argument; (2) 

the generation of data; and (3) the analysis of that data. All three 
items are involved in investigations which test hypotheses while only 
items two and three are used in studies which attempt to answer em- 
pirical questions. 
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THE INHERENT LOOIGAL ARGUMENT is of oruoial impo.-tance when study 

attempts a test of a hypothesis. In effeot, the investigator is trying 

to determine the truth or falsity of his hypothesis. He does this 

through a logical argument described by Polya. ^ It consists of a 

major premise* one or more minor premises, and a conclusion. 

The major premise is typioally a statement which asserts* "If the 

_ ___ hypothesis is a true statement; then ___ events will 

be observed as indicators of that truth." An example of a major prem- 

2 

ise oan be seen in a study reported a few years ago by MoNeil. He 
proposes a hypothesis which asserts that teaohers present different 
instructional treatments for the two sexes of their students. As 
indicators of the truth of that statement he reasoned that boys would 
be nominated more often than girls as recipients of certain kinds of 
teacher aetion. His major premise could be stated as* 

If the hypothesis (teachers provide different instructional 
treatment for boys than they do for girls) is a true state- 
ment; then systematic differences by sex will be seen when 
children are asked to name the students who receive specified 
teacher treatments. 

Two kinds of minor premises have been evolved from Polys's work 
3 

by Raths. The first of these deals with the predicted observation. 

Was it or was it not seen? The premise's exact nature in a given 
study is determined after the data are analyzed. In the McNeil ex- 
ample used above* significant differences by sex were observed. The 
minor premise in that case would be* "There is a systematic sex dif- 
ferientiation in the nominations." 
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The second category of minor premises deals with rival hypotheses, 
rival or alternative explanations for the observation reported in the 
first minor premise. The premise is based upon the recognition that an 
effect in the social sciences often has multiple causes. Once an 
observation has been made, all its possible causes must be examined 
before it can be concluded that the observation supports the truth of 
a specific hypothesis. One of three general conditions might exist 
ranging from no rival hypotheses are apparent to rival hypotheses 
may exist to rival hypotheses are definitely involved. 

The final element of the logical argument is the conclusion. Its 
form in a given study is dependent upon the nature of the two minor 
premises. From the first minor premise comes information as to whether 
or not the truth of the hypothesis being tested is supported. If the 
consequents predicted are observed, support for the truth of the hypo- 
thesis is presented. If the observation is not made, support cannot 
be claimed. (Note: Failure to make the predicted observation does not 

automatically mean rejection of the hypothesis.) The second minor 
premise determines the strength of the conclusion. If rival hypotheses 
are known to be present, very weak support for the truth of the hy- 
pothesis has been developed. If there is the possibility but not the 
probability of rival hypotheses, tentative support is generated. And 
finally, if no rival hypotheses are conceiveable, it is credible that 
the hypothesis is a true statement, 

THE GENERATION OF DATA, the second major facet in profiling, in- 
volves evaluation of three aspects of data generation: units studied; 

treatments experienced by those units; and measurement. If variation 
in any of these three occurs a different set of data are generated. 
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Por example, consider an investigation of the effects of test anxiety 
on achievement. If the study concentrates on a randomly selected group 
of high school seniors as subjects, one set of data will be generated. 

If a group of students who are divergent on a measure of test anxiety 
is selected as subjects. a different set of data will be generated. 

Given a specified group as subjects, variation in the treatment or of 
their experiences will cause different sets of data to be generated. 
Again the test anxiety problem provides an example . One set of data 
could be generated by a treatment in whioh the subjects are given infor- 
mation about the importance of a test and administered a test that is 
constructed for students at a much higher level of education than are 
the subjects. Still a different set of data will be generated if the 
students are repeatedly given a test that is very difficult. If the 
effects of a specific treatment on a specific group are measured by a 
paper and pencil test such as Sarazon’s Test of Test Anxiety, one set 
of data would be generated. On the other hand if the seats in the 
classroom were wired and a galvanic skin response measure were taken, 
quite a different set of data would be generated. 

These three aspects of data generation are displayed graphically 
in Figure 1.^ The scale of unit quality or representativeness runu 
along the dimension QA, treatment quality OC, and measurement quality 
OG. A project which selected a sample perfectly representative of a 
population of interest would be located at Point A, on the cube. If, 
in that same study, a thorough programing of the content and sequence 
of the treatments was employed in generating data, the project would 
be conceptualized as being at Point B. on the quality cube. Fi nally , 
if our study employed perfectly objective, valid, and reliable mea- 
suring techniques, it would be located at Point E. on the cube. 
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FIGURE 1. 

THE RESEARCH QUALITY CUBE 




TREATMENTS Contents & 
Sequenoe 
Programed 



A given study seldom reaches this level of data generation quality* 
Rather it falls somewhere between the extremes. To facilitate profil- 
ing ordinal scales have been developed for these three dimensions as 
shown below. 



Dimensions for the Research Quality Cube 



Representativeness 

= The entire population was studied 
Rjj = Random selection from a specified population was employed 
to determine which units were studied 
R^ = Purposive sampling from a specified population established 
the group studied 
R2 * Volunteers were studied 

R]_ * An unidentified group of subjects was studied 
Treatment 

T^ = A theoretically based treatment was administered and described 
and controls were employed for mediating variables iden- 
tified in the theory AND for variables extraneous to the 
theory that might have an effect. 
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Tcj ■ Same as Tg with the exception of the lack of controls for 
extraneous variables. 

TV ■ Same as Tg with the exception of the lack of controls for 
theory encompassed mediating variables and extraneous 
variables. 

T3 • No theory stated but the employed treatment described in 
detail sufficient for replication. 

T2 ■ Commonly known treatment administered but not described in 
detail. 

Ti - Something of an undescribed nature was experienced by the 
units studied. 

Measurement 



Mcj - Data were generated through the use of either a coimneroially 
standardized or ad hoc instrument AND data are presented 
which establish high validity and reliability for its use in 
this measurement task. 

Mjj ■ Data generated through the use of a commercially standardized 
instrument and evidence presented indicating moderate validity 
and reliability for this application. 

M3 ■ Data generated through a commercially standardized test but 
no evidence presented as to its validity and reliability 
for this application. 

M2 * Data generated through an ad hoc instrument and evidence of 
moderate validity and reliability presented. 

Mi » Data generated through an ad hoc instrument with either no 
supporting evidence as to validity and reliability or evi- 
dence indicating poor validity and reliability on either 
a commercially standardized or ad hoc instrument. 

DATA ANALYSIS PROCEDURES are the final element in profiling. When 
data, typically in the form of numbers, are generated as the supporting 
evidence for a conclusion, understanding of the meaning of those numbers 
is incumbent upon the researcher and the research utilizer. That mean- 
ing is not readily apparent if there is a large quantity of numbers. 
Simplifying procedures have been developed; procedures which are not 
appropriate for all kinds of data. 

The detennination of the correct procedure in a given stuay is not 
an exact science. In developing a procedural flow chart for the profil- 
ing of educational research, sixteen schemes were identified which 
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were supposed to assist in tne selection of the correct statistic for 
given sets of data. Some of these were incomplete schemes in that they 
purported to deal only with limited kinds of statistical analysis.^ 
Some imply a comprehensiveness but fail to be definitive as they list 
a number of statistics appropriate for a given set of conditions. 

Since a single comprehensive grid or table for selecting the 
correct analytic procedure could not be found a second task was under* 
taken. Existing statistical proceaures were catalogued and the assump- 
tions underlying them were listed. An effort to build a comprehensive 
selection procedure by analyzing these items has to this point been 

n 

unsuccessful. (A colleague at Indiana University' has just recently 
attacked this problem using Guttman’s Facet Design and Analysis Tech- 

g 

nique with initially promising results.) 

Because of these problems three grids have been generated for pro- 
filing the data analysis procedures. The first of these deals with 
analytic procedures for sample description. It includes measures of 
central tendency and dispersion and classifies the procedures by 
levels of measurement, i.e., nominal, ordinal, and interval-ratio. 

The second grid is used when an associational analysis is desired. 

It has identical labels for its rows and columns which refer to the 
nature of the measurement on the two variables to be correlated. The 
categories in this case are: 

1. Continuous variables (age, height, I.Q., achievement, etc.) 

2. Forced dichotomy (number of persons over and under 100 I.Q., 
number of persons weighing over and under 150 pounds, etc.) 

3. True dichotomy (student-nonstudent, male-female, etc.) 

Given the nature of the two variables on which an associational anal- 
ysis is desired the grid can be used to select the appropriate 
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statistio. Four speoial cases exist and are shown with the grid. 

Three of these are instances in which more than two variables are in- 
volved. The final case covers correlation among ordinal variables. 

The third grid deals with inferential statistics, instances in 
which a generalization about the relationship between the numbers 
generated by observation of some sample are indicative of observations 
that could be made on the entire population. The categorizing elements 
on this grid are the number of dependent and independent variables, 
the level of measurement, and the number of groups. Again the determi- 
nation of the appropriate level on each category for a given set of 
data leads to the recommended statistic. 

The use of these grids leads to a specific statistic (in the infer- 
ence grid there is the possibility of alternatives). Through the 
article the analytic procedure actually used can be identified. Two 
quality categories follow from a comparison of the statistic used and 
the statistic appropriate for the data and purpose of the study: first, 

the statistic used is identical with the statistic identified as ap- 
propriate; second, they are different. In the former the research is 
profiled as appropriately analyzed; in the latter, as inappropriately 
analyzed . 

PROFILING SUMMARIZED: When a study has been analyzed and profiled, 
it has been described on the following basis: 

A. Is it (l) a test of a hypothesis, or (2) an answer to an 
empirical question? 

A . If it is a test of a hypothesis, is the strength of conclu- 
sion: I The hypothesis is very little more credible; II 

more credible; or III very much more credible? 
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B. What Is the quality of the data generation procedure 

C. Ia the data: (a) appropriately analyzed; or (b) inapprop- 
riately analyzed. 

It should be noted that a single project may consist of several sub- 
studies, eaoh of which may be profiled separately. A decisional flow 
chart has been developed for arriving at the profile for a given study. 
It is appended. Your reactions regarding its adequacy are welcomed. 

It is believed that through profiling completed research their 
adequacies and inadequacies can be made apparent and can more readily 
be considered as the conclusions of the research are weighed in de- 
cision situations. One further benefit is seen. Studies of such pro- 
files should pinpoint problems that could keep research methodologists 
busy for years to come. 
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REPORT TITLE: 



AUTHOR 



SOURCE: 



STOP The report Is either not research or It is an Incomplete part of the research 
process. 

CO LOGIC 

_ q inswr to an kpirlol Qtttstloa 



Test of a Hypothesis 
_ H s Stop, Illogical relationship. 

H No conclusion. 

" H" c Hypothesis is questionable. 

" (Rival hypotheses must be considered a cause of the consequents) 

H Hypothesis is credible. 

(Rival hypotheses may be considered a cause of the consequents) 

__ H Hypothesis is verified. 

(Rival hypotheses cannot be considered as a cause of the consequents) 

fry DATA QUALITY - REPRESENTATIVENESS 

_ R X An unidentified group of subjects was studied. 

" R 0 Volunteers were studied. 

" r^ Purposive sampling from a specified population established the group studied. 

r Random selection from a specified population established the group studied. 

~ R 5 The entire population was studied. 

HP) DATA QUALITY - TREATMENT 

T No theory; something undefined happened to the units studied. 

~ xi No theory; treatment description incomplete, or detailed elsewhere. 

To No theory; treatment described in detail in the report. 

~ Ta Theory stated but no controls on varibles. 

Tc Theory stated and mediating variables controlled. 

* T^ Theory stated, mediating variables controlled, and techniques used to distribute 

* possible extraneous variances. 
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DATA QUALITY - MEASUREMENT 

Available information indicates instrument is Invalid. . .. 

Project Developed Instrument with low validity (V), reliability (R), objectivity 
( 0 ), or other instrument with no Info about validity or data source. 

Used Commercially Produced or Cther-Projec.t Developed instrument with low 



V,R,0 for this application. , , _ 

Used Project Developed instrument or Other-Project Developed Instrument with 

moderate V,R,0, for thla application. 

Used instrument which was Project Developed with high V,R,Q, or Other-Project 
developed with high V,R,0, or Commercially Produced with moderate ^,R,0 for 
this application. 

Used Commercially Produced instrument with high V,R,0 for tuis application. 



IX> STATISTICAL ANALYSIS 

_ a Appropriately analyzed 
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(Attachment B - continued) 

I D3 

LITERATURE SEARCH ACTIVITY DETAIL 

The statements below are keyed by numbers to the activities in 

the TRDPR Literature Search Flow Chart. They are not necessarily in 

numerical sequence. 

1. Start. 

2. Reviewer-Evaluators Selected— Several persons must be employed to do 

the work of reading and evaluating the separate items of related litera- 
ture. It is suggested that these persons be recent doctoral graduates with 
a composite of specialities likely to be encountered in the literature 
(e.g. : perception, vision, reading, learning, etc.). 

3. Inclusion-Exclusion Criteria Drafted—The project personnel need to 
prepare an initial statement of criteria for the inclusion or exclusion 
of separate pieces of literature in this synthesis activity. This state- 
ment is to be used as a working draft with the project review panel. 

4. Review Panel Selected— Specialists from each of the disciplines likely 
to contribute information about models or models themselves should be 
contracted with as an advisory /review panel for the project. (Possible 
participation should be arranged prior to submitting a proposal and the 
tentative advisory panel members and their qualifications listed with the 
proposal.) 

5. Source Lists Prepared--The project staff should prepare a draft listing of 
the literature sources to be examined in the project. 

6. Scarce Lists Approved--The Advisory Panel should check the draft listing of 
literature sources. Where possible they should add additional sources to insure 
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comprehensiveness of the literature search. Where the Advisory Panel 
is certain of the unproductiveness of a soit. ce, it should be deleted. 

The source list that is approved will provide the boundaries for the 
literature search in terms of published works and unpublished research 
reports and articles to be examined. 

7. I &E Criteria Approved--The Advisory Panel should check the draft 
criteria for their clarity and comprehensiveness and made modifications 
where their expert judgment deems it warrented. This activity will 
produce a set of criteria on which to base the decision to include (or 
exclude) a specific document or article in the literature synthesis. 

8. Reviewer-Evaluators Trained— The selected reviewer-evaluators must 
understand (1) the Inclusion-Exclusion Criteria, (2) the overall purpose 
of the search (see items 19 & 37-42) , (3) the procedures to be employed 
in the search, (4) the nature of and format for abstracts on all processed 
literature items, (5) the research profiling approach to the evaluation of 
the methodological adequacy of completed research (See Occasional Paper No. 3, 
"Profiling Educational Research," attached.), and (6) resources available 

to them for resolving questions encountered in their work. 

9. Listed Sources Located- -After a literature source list has been approved, 
each item in it must be physically located. If the originals of complete 
copies can be obtained they should be. If not, arrangements must be made 
to work with them at their current location. 

10. Items in Listed Sources Read— One or more of the reviewer-evaluators must 
read each document (report or article) from each listed source and examine 
it against the inclusion-exclusion criteria. 
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11. Items Included Catalogued— Each document which satisfies the inclusion 
criteria shall be catalogued by recording source, bibliographic information, 
and topics dealt with. 

12. Abstracts Written When Needed— A brief abstract for all included items 
shall be prepared giving general description of the item and explicit 
listing of assumptions, hypotheses, conclusions, and models described. 

13. Items Excluded Catalogued and Stored— Each document that fails to meet 
the inclusion criteria shall be catalogued by recording source, biblio- 
graphic information and reason for exclusion. 

14. Each Model Catalogued— Every model discussed in the literature will be 
catalogued by recording a brief description of the model and Its biblio- 
graphic source. 

15. Assumptions Listed— A separate file will be kept which lists the 
assumptions made by an author or researcher. This listing will include 
a statement of the assumption and bibliographic information. 

16. Models Analyzed for Elements— Each model will be analyzed to identify 
their constituent elements. 

17. Common Models Merged— When two or more separately referenced models are 
analyzed as consisting of the same component elements, they will be 
merged and a statement made giving the rationale for this merger. 

18. Each Different Model Described— As complete a description as possible 
will be prepared for each model remaining on the lists after all mergers 
have been effected. 

19. Tests Specified to Confirm Merger of Models— The merger of two or more 
models will have been done on the basis of a logical analysis of their 
elements. This logic must be tested empirically in Phase II of the 
Program. Thus, as models are merged, empirical tests which will confirm 
or dlsconflrm each merger must be designed. 
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20. Progress Report Approved by Advisory Panel— The Advisory Panel will be 
asked at this point to check (1) the adequacy of the analysis of each 
Identified model, (2) the logic of the merger of two or more models, 

(3) the comprehensiveness of the work to date, l.e., are there any 
models known to members of the Panel that have not been Incorporated 
Into the listings and synthesis to date. 

21. Items Sorted by Research and Non-Research —As each document Is read 
It Is to be catalogued either as a report of empirical research or 
other. If the report poses and empirically tests a hypothesis or 
answers an empirical question it is to be accepted as research. 

22. Untested Hypotheses Listed— All documents labeled non-research will 
be examined for untested hyotheses. These will be listed along with 
untested hypotheses presented In research documents. Along with the 
hypothesis, bibliographic information will be recorded. 

23. Research Items Prof lied— Each document labeled a research item in No. 21 
will be analyzed for methodological characteristics and adequacy. This 
analysis will examine the inherent logic in the research design, the 
quality of sampling, measurement, and treatment techniques, and the 
adequacy of the statistical analysis employed. (See Occasional Paper 
No. 3, "Profiling Educational Research", attached). 

24. Research Conclusions Catalogued --Each conclusion in these articles 
will be listed along with a profile of the adequacy of the research 
methodology on which it is based. Common conclusions will be merged 
and the resultant conclusions listed under two rubrics, methodologically 



sound or unsound 
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25. Methodologically Sound Conclusions Checked for Contradictions— -The 
listing of conclusions based on sound research methodology must be 
examined for contradictions. Where two or more contradictory con- 
clusions are identified, they must be considered as inconclusive and 
sorted from what is known about the phenomenon under study. 

26. Methdologically Weak Conclusions Listed--All conclusions based on questionable 
research methods shall be listed as still-to-be-tested hypotheses along 

with bibliographic references and the rationale for their placement in 
this category. 

27. Contradictory Items Merged with Weak Conclusions and Untested Hypotheses— 

The conclusions listed in activity 25 and 26 and the untested hypotheses 
listed in activity 22 are to be accumulated and common items merged to 
form a list of untested hypotheses. 

28. Assumptions and Sound Conclusion Lists Checked for Duplication— The 
listings from activities 15 and 25 are to be compared for duplication. 

Any items common to the two lists are to be deleted from the list of 
assumptions. 

29. Sound Conclusions Listed— This listing constitutes the empirically 
substantiated body of knowledge about the phenomenon being studied. 

30. Assumptions Listed— Two categories of assumptions exist in research, 
things beyond our ability to empirically confirm and things that 
can be but have not been confirmed to date. Any final list of 
assumptions should not include any of the second type if they have 
been empirically confirmed by other research efforts. The list of 

assumptions possible after activity 28 will exclude any such items. 
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31. Testable Assumptions Listed— Those assumptions In the listing produced 
in activity 30 which are subject to empirical tests shall be separately 
listed along with suggestions for the nature of those empirical tests. 

32. Merged with Untested Hypotheses— The assumptions which are subject 
to empirical tests are comparable to untested hypotheses and should 
be merged with the list produced in activity 27. 

33. Conclusions and Untestable Assumptions Merged with Model Descriptions 
Where Appropriate— This activity produces a synthesis of what is known 
and assumed for each separate model identified in the literature search. 

As such it is the most detailed statement about each model of the 
phenomenon being studied conslstant with the current state of the art. 

34. Progress Report Approved by Project Advisory Panel— The synthesis 

of what is known and assumed with accepted model descriptions should 
be examined by the Panel for logic and comprehensiveness. If it appears 
possible to further merge either partial models into a comprehensive 
model or to merge comprehensive models, such mergers should be discussed 
with and approved by the Advisory Panel. 

35. Items Related to Models— The listing of untested hypotheses should 
be culled for items that relate to one or more of the models approved 
in activity 34. 

36. Items Not Related to Models— Those items remaining in the list of untested 
hypotheses after activity 35 should be catalogued along with biblio- 
graphic references, and suggestions for the nature of their empirical 
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37. Models Examined and Untested Hypotheses Regarding Structure Listed — 
Hypotheses about the structure of models approved in activity 34 
should be listed by (a) examination of each model itself or (b) from 
the lists generated in activity 35. Where possible the details for 
testing a specific hypothesis should be specified and documented. 

38. Models Examined and Undefined Constructs Listed— It is anticipated 
that hypothetical constructs will be involved in the models that 
reach this stage. Those constructs should be listed and research 
proposed which will empirically define them (l.e., attention, motiva- 
tion, etc.). 

39. Tests Specified for Developing Each Model's Calculus-of-Operation— 

It is anticipated that models which reach this stage will be verbal 
or verbal -pictorial representations of the phenomenon studied. In 
such a model the interrelationship of elements is suggested but not . 
specified. Tests must be described which will quantify these elements 
(l.e., eye-movement, attention, meaning, etc.) and develop mathematical 
formulations for their interrelationships. 

40. Tests Specified to Determine Each Model's Predictive Capability— A model 
is a representation of some thing or phenomenon. It displays the 
elements of the modeled phenomenon, the manner in which those elements 
Interact, and the results of their interaction. When a good model is 
operated it should produce results that are produced by the phenomenon 
itself. Each model identified in the literature search and approved 

in activity 34 should be examined for this predictive capability. For 
models that have reached a sufficient stage of development, tests of 
predictive capability must be specified. 
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41. Progress Report Approved by Project Advisory Panel--This approval 

covers five categories of proposed research efforts: (1) Studies 

to confirm the merger of two or more models; (2) Tests of hypotheses 
about the structure of Identified models; (3) Studies designed to 
further define constructs In the models; (4) Studies designed to 
facilitate quantification of the factors Involved In the models 
and of their function; and (S) Studies designed to test a model's 
predictive capability. The Advisory Panel should examine the specific 
studies proposed for their logic and comprehensiveness and make 
appropriate suggestions for improvement. 

42. Report— The final report shall Include: (1) a description of each 

model Identified and approved by the Advisory Panel along with docu- 
mentation of Its features at currently known and its operational 
characteristics; (2) Proposals for specific studies recommended for 
the further development of these models; (3) A listing of the untested 
hypotheses which are unrelated to any model; and (4) an annotated 
bibliography of those documents examined but rejected for this literature 
synthesis. 
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Targeted R&D Program on Reading 
Project No. 3: Status Survey 



I. Statement of Purpose : 

The Targeted R&D Program on Reading assumes the existence of a 
significant, quantifiable deficit between the present state of reading 
ability in the U.S. and the individual and social literacy needs of the 
populace. The Status Survey specified in this Section of this Request 
for Proposal involves a search of literature relevant to the pursuit of 
the three tasks listed below. Using existing survey and test data, and 
other scientific literature: 

1. Determine the extent and distribution of the national 
"reading problem." 

2. Determine the use frequency apd use distribution of 
instructional methods, approaches, procedures, materials, 
and equipment for reading instruction. 

3. Describe the nature and extent of current practice in 
teacher training for (2) above. 

II. Definitions : 

1. R eading achievement : The phenomenon quantified by existing, norm- 

referenced reading tests which are in widespread use. Reading 
achievement and reading ability are used synonymously in this 



document. 
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2. Method of reading Instruction : A systematic collection of 

instructional materials, equipment, and guidance for teachers who 
use the materials and equipment to help children develop reading 
achievement . 

3. Approach to reading instruction : A generic term descriptive of 

families of "methods." Examples of approaches are: "a basal 

reader approach," "a code-emphasis approach," "a language exper- 
ience approach," and "a meaning-emphasis approach." 

4. Instructional materials : The books, worksheets, and other software 

used in reading instruction. 

5. Equipment : The computers, pacers, tachistoscopes, and other hardware 

used in reading instruction. 

6. Conceptual basis : The empirically and/or intuitively derived 

information upon which an instructional method is built. 

III. Scope of Work : 

1. Determine the extent and distribution of the National "reading problem.” 
The Contractor shall provide 100 copies of a report which identifies, 
analyzes, and summarizes existing survey and test data which relate to 
the reading ability of various population groups found in the U.S. The 
Contractor’s full insight is sought in designing and producing the 
synthesis of these items in a manner which specifies the levels of reading 
achievement currently being attained, according to: 

a. Learner characteristics such as sex, age, race, I.Q., etc; and 

b. Characteristics of the learner's environment such as socioeconomic 
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level, geographic area, language spoken in home, level of 
of formal education attained by parents, etc. 

2 . Determine the use frequency and use distribution of instructional 
methods* approaches, procedures, materials, and equipment for 
reading instruction . 

It is assumed that a wide variety of instructional methods, 
approaches, materials, and equipment is actively used in school settings 
and other settings to teach people to read. Further, it is assumed, 
on advice from experts and researchers who study reading, that the 
fundamental differences among the conceptual bases behind the many 
instructional approaches, materials, and equipment have not been 
determined. The Contractor shall provide 100 copies of a report which 
details the findings of a literature search designed to pursue the 
following questions: 

a. What methods, materials, approaches, equipment, and 
procedures are used to teach reading in the United States, 
and to what extent are the major items in these categories 
used; i.e., how many learners are exposed to them? 

b. Which methods of reading instruction are built upon 
essentially different pools of basic knowledge? 

(This question assumes that although methods will differ 
widely in matters related to formats for presenting learning 
sequences, relatively few methods use intuitive and/or 
empirical knowledge not used by other methods. 

c. How much time and resources are expended directly upon 
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developmental reading instruction and remedial reading instruction, 
and indirectly upon the supervision of reading instruction? 
d. What relationships between (2b) and (la and b) can be shown? 

3. Describe the nature and extent of current practice in the training of 
those who teach reading . 

In the United States, reading ability is the product of an extremely 
complex system of institutions and people. Although the detail of the 
system varies from locality to locality, its gross dynamics may be 
charted as follows: 



Traditional public 
mandate for 
reading instruction 



T_ 



[Certification Boardsl 

X' 



Institutions which train administrators 
and teachers (graduation and certificate 
requirements) 

v ' i 

“7 [ Sc h ool Administra tors and Teachers _j 

__ 4 ** ■. 

> lActual Practice of Read ing In st ruction! 

\U 

Change in the re ading behavior of j 



children and adults 



Every arrow in the flow chart identifies a point at which distortion 
can occur: Do certification boards accurately translate the public 

mandate into teacher training requirements? Do teachers practice as 
they were taught to practice? Every block in the flow chart identifies 
a point at which an information-exchange activity, in varying quality 
and quantity, takes place. The Contractor shall provide 100 copies of 
a report which details the findings of a literature search designed 
to describe the system charted above in terms of the discrepancies 
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between announced or implicit objectives and actual performance at 

I 

each level. To the extent that information cueists in the published 
literature the scope of the studies shall include (a) all 4-year 
institutions which train at least 100 elementary school teachers 
per year; (b) certification standards for elementary school teachers, 
reading specialists, and reading supervisors in the 50 States and 
the District of Columbia; and (c) teacher performance in relation 
to (a) and (b). 

IV. Plan of Operation : 

Appended to the Section of this Request for Proposal titled "Project 
No. 2: Literature Search" are (1) a flow chart and \ explanatory key; and 

(2) "Occasional Paper No. 3: Profiling Educational Research" by 

William J. Gephart. These two items suggest the level of effort that 
will be required in carrying out "Project No. 3: Status Survey", as 

well. It is recognized that the flow chart requires adaptation to the 
requirements of "Project No. 3: Status Survey." Bidders are expected 

to present a documented logical case to justify the" procedures that they 
propose to follow, whether those procedures adopt, modify, or replace 
those shown in the flow chart and occasional paper. Regardless of the 
procedures proposed, bidders will be expected to provide a comprehensive 
listing of information sources that will be used. 

V. Criteria for the Evaluation of Proposals ; 



1. Evidence that the Offeror can provide the overall management 
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capability necessary to Insure quality, completeness, and 
efficient utilization of time and funds. 

2. Appropriateness of the training and experience of the staff 
and consultants. 

3. Appropriateness and adequacy of the technical methodology 
proposed for each of the three activities In the section 
labeled "Scope of Work." 
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HEW 's Office of Education announced today the launching of a new research 
program in connection with the National Right to Read Effort for the 1970' s 
endorsed by President Nixon in his educational reform message. 

"The Targeted Research and Development Program on Reading is designed to 
provide the scientific foundation for the Right to Read Effort," said Acting 
U.S. Commissioner of Education Terrel H. Bell. "The goal of this important 
program is to enable every child in a national sample to achieve sufficient 
reading skill by age 10 to become a competent adult reader." 

More than $500,000 has been awarded by the Office of Education's National 
Center for Educational Research and Development (NCERD) ' a support three initial 
research projects during their first six months of operation. Provided satis- 
factory progress is made, NCERD later plans to award an additional $1 million 
to continue the projects which will run from 12 to 30 months. 

Contracts (subject to final negotiations) went to: Educational Testing 

Service (ETS) Princeton, N.J. , $338,099; Rutgers, The state University, Brunswick, 
N. J. , $137,144; Educational Testing Service (ETS Western Office), Berkeley, 

Calif., $97,747. 

The first project, headed by Donald A. Trismen at ETS (Princeton) , will 
develop standards for adult reading competence by pinpointing the basic printed 
materials with which people presently must work in order to perform tasks 
neepssary in our society, in addition, it will provide tools for evaluating 
reading instruction systems and devise a method for determining the progress 
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being made toward the goal of eliminating reading failure. 

Martin Kling of Rutgers, director of the second project, will search 
the scientific literature to identify promising lines of investigation and 
determine the work needed to produce valid models of the reading process, 
the learning-to-read process, and of language acquisition. 

In the third project, Reginald A. Corder at ETS (Berkeley) will use 
current reading statistics to produce a profile of reading achievement in 
the u.S. by age, sex, ethnic background, and other groupings. He will also 
catalog the various techniques, equipment, procedures, and practices used 
in teaching reading. 

Next summer, the completion of projects two and three will permit 
further refinement of plans for the next stage of the program. This stage 
will support research to increase scientific knowledge .about reading while 
advancing work on promising instruction programs. 

The projects announced today were selected from among 31 bids received 
in response to a request for proposals issued by NCERD. Panels of experts 
from within and outside the agency reviewed the proposals. 

Support for the projects is provided under the amended Cooperative 
Research Act which authorizes research, development, and dif ni nation to 
improve education at all levels. 
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